{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "zkufh760uvF3"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zkufh760uvF3"
      },
      "source": [
        "![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/nlu/blob/master/examples/colab/Training/binary_text_classification/NLU_training_sentiment_classifier_demo_twitter.ipynb)\n",
        "\n",
        "\n",
        "\n",
        "# Training a Sentiment Analysis Classifier with NLU\n",
        "## 2 class twitter classifier training\n",
        "With the [SentimentDL model](https://nlp.johnsnowlabs.com/docs/en/annotators#sentimentdl-multi-class-sentiment-analysis-annotator) from Spark NLP you can achieve State Of the Art results on any multi class text classification problem\n",
        "\n",
        "This notebook showcases the following features :\n",
        "\n",
        "- How to train the deep learning classifier\n",
        "- How to store a pipeline to disk\n",
        "- How to load the pipeline from disk (Enables NLU offline mode)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dur2drhW5Rvi"
      },
      "source": [
        "# 1. Install Java 8 and NLU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hFGnBCHavltY"
      },
      "source": [
        "!pip install -q johnsnowlabs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f4KkTfnR5Ugg"
      },
      "source": [
        "# 2. Download twitter Sentiment dataset\n",
        "https://www.kaggle.com/cosmos98/twitter-and-reddit-sentimental-analysis-dataset\n",
        "#Context\n",
        "\n",
        "This is was a Dataset Created as a part of the university Project On Sentimental Analysis On Multi-Source Social Media Platforms using PySpark."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OrVb5ZMvvrQD"
      },
      "source": [
        "! wget https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/classifier-dl/reddit_twitter_sentiment/Twitter_Data.csv\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "y4xSRWIhwT28",
        "outputId": "cb744bdd-bf86-4bec-b200-e8249a2e3f1c"
      },
      "source": [
        "import pandas as pd\n",
        "train_path = '/content/Twitter_Data.csv'\n",
        "\n",
        "train_df = pd.read_csv(train_path)\n",
        "# the text data to use for classification should be in a column named 'text'\n",
        "# the label column must have name 'y' name be of type str\n",
        "train_df = train_df.rename(columns={'clean_text': 'text'})\n",
        "\n",
        "columns=['text','y']\n",
        "train_df = train_df[columns]\n",
        "train_df"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                  text         y\n",
              "0    new post added mumbai press official site prod...  positive\n",
              "1    not wrong the actual temperature might but and...  positive\n",
              "2    why pakistan crying name modi every day how na...  negative\n",
              "3    congress years wasnt able complete one rafale ...  positive\n",
              "4    public toilet near kanagadurga temple nizampet...  positive\n",
              "..                                                 ...       ...\n",
              "595                    jai hind modi very nice thought  positive\n",
              "596  after going thru all the comedy speeches shri ...  positive\n",
              "597  mistry man not then why drag modi the nri foll...  negative\n",
              "598  why modi have not held single press conference...  negative\n",
              "599  modi government which fails protect its women ...  negative\n",
              "\n",
              "[600 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a83c933f-79ee-4f37-adbf-ee063ac69606\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>y</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>new post added mumbai press official site prod...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>not wrong the actual temperature might but and...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>why pakistan crying name modi every day how na...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>congress years wasnt able complete one rafale ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>public toilet near kanagadurga temple nizampet...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>595</th>\n",
              "      <td>jai hind modi very nice thought</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>596</th>\n",
              "      <td>after going thru all the comedy speeches shri ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>597</th>\n",
              "      <td>mistry man not then why drag modi the nri foll...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>598</th>\n",
              "      <td>why modi have not held single press conference...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>599</th>\n",
              "      <td>modi government which fails protect its women ...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>600 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a83c933f-79ee-4f37-adbf-ee063ac69606')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a83c933f-79ee-4f37-adbf-ee063ac69606 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a83c933f-79ee-4f37-adbf-ee063ac69606');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-806e5a7a-a70b-44c5-9c32-1aad413d113f\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-806e5a7a-a70b-44c5-9c32-1aad413d113f')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-806e5a7a-a70b-44c5-9c32-1aad413d113f button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0296Om2C5anY"
      },
      "source": [
        "# 3. Train Deep Learning Classifier using nlu.load('train.sentiment')\n",
        "\n",
        "You dataset label column should be named 'y' and the feature column with text data should be named 'text'"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "3ZIPkRkWftBG",
        "outputId": "fbb3d3a1-9cf5-4ecc-89a4-fdb6e0d59d17"
      },
      "source": [
        "from sklearn.metrics import classification_report\n",
        "from johnsnowlabs import nlp\n",
        "# load a trainable pipeline by specifying the train. prefix  and fit it on a datset with label and text columns\n",
        "# by default the Universal Sentence Encoder (USE) Sentence embeddings are used for generation\n",
        "trainable_pipe = nlp.load('train.sentiment')\n",
        "fitted_pipe = trainable_pipe.fit(train_df.iloc[:50])\n",
        "\n",
        "# predict with the trainable pipeline on dataset and get predictions\n",
        "preds = fitted_pipe.predict(train_df.iloc[:50],output_level='document')\n",
        "#sentence detector that is part of the pipe generates sone NaNs. lets drop them first\n",
        "preds.dropna(inplace=True)\n",
        "print(classification_report(preds['y'], preds['sentiment']))\n",
        "\n",
        "preds"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Warning::Spark Session already created, some configs may not take.\n",
            "sent_small_bert_L2_128 download started this may take some time.\n",
            "Approximate size to download 16.1 MB\n",
            "[OK!]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/nlu/pipe/pipeline.py:149: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  dataset.y = dataset.y.apply(str)\n",
            "/usr/local/lib/python3.10/dist-packages/nlu/pipe/utils/data_conversion_utils.py:160: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  data['origin_index'] = data.index\n",
            "/usr/local/lib/python3.10/dist-packages/nlu/pipe/utils/data_conversion_utils.py:160: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  data['origin_index'] = data.index\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "    negative       0.00      0.00      0.00        19\n",
            "    positive       0.62      1.00      0.77        31\n",
            "\n",
            "    accuracy                           0.62        50\n",
            "   macro avg       0.31      0.50      0.38        50\n",
            "weighted avg       0.38      0.62      0.47        50\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/nlu/pipe/extractors/extractor_methods/base_extractor_methods.py:356: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  df[cols_to_explode] = df[cols_to_explode].apply(pad_same_level_cols, axis=1)\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                             document  \\\n",
              "0   new post added mumbai press official site prod...   \n",
              "1   not wrong the actual temperature might but and...   \n",
              "2   why pakistan crying name modi every day how na...   \n",
              "3   congress years wasnt able complete one rafale ...   \n",
              "4   public toilet near kanagadurga temple nizampet...   \n",
              "5   the foundation for new india 2022 has already ...   \n",
              "6   dear governorani can you let the people indian...   \n",
              "7   this daft donkey’ dick aap was born the iac mo...   \n",
              "8   major reason for social hatred and strife modi...   \n",
              "9   demo was black money caught modi did inspite r...   \n",
              "10               one the best ministers modis cabinet   \n",
              "11  raghuram rajan sent list high profile bank fra...   \n",
              "12  governor kalyan singh aligarh 23rd march all a...   \n",
              "13  this campaign low hanging fruit seems giving f...   \n",
              "14  strict policing that too health system sir reg...   \n",
              "15  shatrughan sinha was far bigger public figure ...   \n",
              "16  people wish your vision india and least intere...   \n",
              "17  chowkidar hee chor hain baap chor beta bada ch...   \n",
              "18  with modi all his drawbacks atleast know what ...   \n",
              "19  compare that with modi’ 2014 “vikas purush” el...   \n",
              "20  with welfare delivery gst ibc and feo place mo...   \n",
              "21  young and dynamic chowkidar says you are not w...   \n",
              "22  dont forget petrol prices have risen ₹ modi go...   \n",
              "23  prime minister narendra modi has urged voters ...   \n",
              "24  when someone asks random question economy sche...   \n",
              "25  from whatsapp have two options this elections ...   \n",
              "26  heres interesting section work which gives com...   \n",
              "27  know why you willpapu his chamchas nothing tar...   \n",
              "28  only rahul gandhis politics love can defeat th...   \n",
              "29  one see the fake calculation calsi chek kariye...   \n",
              "30  being born religion where female deities worsh...   \n",
              "31  narendra modi became and despite biased viciou...   \n",
              "32  think hindus should back off and let them suff...   \n",
              "33  from the very beginningmodi doing wada faramos...   \n",
              "34  women are powerful nature wat hinduism states ...   \n",
              "35  almost 4000 crore spent but the ganga more pol...   \n",
              "36  seriously must have sick mind call small kid r...   \n",
              "37  agree with you was unrequired was kinda uncomf...   \n",
              "38  rajdeep think 4got that imran not indian never...   \n",
              "39  the great modi trap ways congress has walked into   \n",
              "40  has come new meanings nationalism hindu and su...   \n",
              "41               modi govt hindus are behaving wildly   \n",
              "42  sir one request why bjp candidate contesting f...   \n",
              "43  vapas lana hai desh agey badana hai vote for m...   \n",
              "44  even with massive mandate 336 seats the nda go...   \n",
              "45  can promise what can delivered epf pension uni...   \n",
              "46  and even print this seriously whats this elect...   \n",
              "47  she has asked three questions from modi and he...   \n",
              "48  dear all tsunami favour modi 2019 coming from ...   \n",
              "49  rahul gandhis politics love can defeat the mod...   \n",
              "\n",
              "                 sentence_embedding_small_bert_L2_128 sentiment  \\\n",
              "0   [-0.2672112286090851, 0.22553667426109314, -0....  positive   \n",
              "1   [-1.243513822555542, 0.24190887808799744, -0.4...  positive   \n",
              "2   [-0.7444853782653809, -0.0514342300593853, -0....  positive   \n",
              "3   [-0.34242647886276245, 0.46881920099258423, -0...  positive   \n",
              "4   [-1.1381851434707642, 0.512217104434967, -0.74...  positive   \n",
              "5   [-0.4057338237762451, 1.0029019117355347, -0.9...  positive   \n",
              "6   [-1.4633989334106445, 0.0006002967129461467, -...  positive   \n",
              "7   [0.04606145992875099, 0.3098487854003906, 0.02...  positive   \n",
              "8   [-0.9470604658126831, 0.27183642983436584, -1....  positive   \n",
              "9   [-0.7136786580085754, 0.0788763239979744, -0.5...  positive   \n",
              "10  [-0.3664279282093048, 0.3727397918701172, -0.4...  positive   \n",
              "11  [-1.0506610870361328, 0.20963071286678314, -0....  positive   \n",
              "12  [-0.20803016424179077, 0.07477151602506638, -0...  positive   \n",
              "13  [-1.2681258916854858, 0.24513696134090424, -0....  positive   \n",
              "14  [-0.9757605195045471, 1.0792640447616577, -0.4...  positive   \n",
              "15  [-0.5276148319244385, 0.2546652853488922, -0.1...  positive   \n",
              "16  [-1.0698672533035278, 0.5003032088279724, -0.4...  positive   \n",
              "17  [-0.5458223223686218, -0.23064681887626648, -0...  positive   \n",
              "18  [-0.8066104054450989, -0.014454682357609272, -...  positive   \n",
              "19  [-0.5401442646980286, -0.41557249426841736, -0...  positive   \n",
              "20  [-0.5764278769493103, 1.2036645412445068, -1.0...  positive   \n",
              "21  [-0.08391488343477249, -0.43613195419311523, 0...  positive   \n",
              "22  [-0.10981855541467667, 0.5448974370956421, -0....  positive   \n",
              "23  [0.27992355823516846, 0.0582934208214283, -0.2...  positive   \n",
              "24  [-0.5603336691856384, -0.1663953959941864, -0....  positive   \n",
              "25  [0.02258816733956337, -0.18138407170772552, -0...  positive   \n",
              "26  [-1.3662021160125732, 0.042950600385665894, 0....  positive   \n",
              "27  [-1.0259658098220825, 0.31534343957901, -0.242...  positive   \n",
              "28  [-0.411965548992157, -0.5224093198776245, -0.6...  positive   \n",
              "29  [-1.0891247987747192, -0.5220181345939636, -0....  positive   \n",
              "30  [-1.0681140422821045, -0.6835974454879761, -0....  positive   \n",
              "31  [-0.20442193746566772, -0.3683846890926361, -0...  positive   \n",
              "32  [-0.9112239480018616, 0.17845268547534943, -0....  positive   \n",
              "33  [-1.3499250411987305, 0.23698307573795319, -0....  positive   \n",
              "34  [-0.47838863730430603, 0.19593238830566406, -0...  positive   \n",
              "35  [-1.229308009147644, -0.07380063086748123, -0....  positive   \n",
              "36  [-0.8348453640937805, -0.03178590536117554, -0...  positive   \n",
              "37  [-0.4827183485031128, 0.08657485246658325, -0....  positive   \n",
              "38  [-0.6942201852798462, -0.3642423152923584, -0....  positive   \n",
              "39  [-0.9660191535949707, -0.2219739854335785, -0....  positive   \n",
              "40  [-0.03318127244710922, 0.04329724237322807, 0....  positive   \n",
              "41  [-0.5563615560531616, 0.4725855588912964, 0.10...  positive   \n",
              "42  [-0.6955257654190063, 0.37047961354255676, -0....  positive   \n",
              "43  [-0.8704789876937866, -0.22370557487010956, -0...  positive   \n",
              "44  [-0.20755957067012787, 0.34972018003463745, -0...  positive   \n",
              "45  [-1.3415300846099854, 1.6326956748962402, -0.6...  positive   \n",
              "46  [-1.4377732276916504, -0.11478081345558167, 0....  positive   \n",
              "47  [-1.193617343902588, -0.02149897627532482, 0.1...  positive   \n",
              "48  [-0.5854026675224304, -0.21378959715366364, -0...  positive   \n",
              "49  [-0.39933672547340393, -0.5969381332397461, -0...  positive   \n",
              "\n",
              "   sentiment_confidence                                               text  \\\n",
              "0                   8.0  new post added mumbai press official site prod...   \n",
              "1                   8.0  not wrong the actual temperature might but and...   \n",
              "2                   2.0  why pakistan crying name modi every day how na...   \n",
              "3                   2.0  congress years wasnt able complete one rafale ...   \n",
              "4                   1.0  public toilet near kanagadurga temple nizampet...   \n",
              "5                   7.0  \\nthe foundation for new india 2022 has alread...   \n",
              "6                   3.0  dear governorani can you let the people indian...   \n",
              "7                   6.0  this daft donkey’ dick aap was born the iac mo...   \n",
              "8                   5.0  major reason for social hatred and strife modi...   \n",
              "9                   1.0  demo was black money caught modi did inspite r...   \n",
              "10                  1.0              one the best ministers modis cabinet    \n",
              "11                  8.0  raghuram rajan sent list high profile bank fra...   \n",
              "12                  1.0  governor kalyan singh aligarh 23rd march all a...   \n",
              "13                  3.0  this campaign low hanging fruit seems giving f...   \n",
              "14                  1.0  strict policing that too health system sir reg...   \n",
              "15                  6.0  shatrughan sinha was far bigger public figure ...   \n",
              "16                  1.0  people wish your vision india and least intere...   \n",
              "17                  7.0  chowkidar hee chor hain baap chor beta bada ch...   \n",
              "18                  7.0  with modi all his drawbacks atleast know what ...   \n",
              "19                  2.0  compare that with modi’ 2014 “vikas purush” el...   \n",
              "20                  7.0  with welfare delivery gst ibc and feo place mo...   \n",
              "21                  2.0  young and dynamic chowkidar says you are not w...   \n",
              "22                  5.0  dont forget petrol prices have risen ₹ modi go...   \n",
              "23                  1.0  prime minister narendra modi has urged voters ...   \n",
              "24                  1.0  when someone asks random question economy sche...   \n",
              "25                  2.0  from whatsapp have two options this elections ...   \n",
              "26                  1.0  heres interesting section work which gives com...   \n",
              "27                  6.0  know why you willpapu his chamchas nothing tar...   \n",
              "28                  2.0  only rahul gandhis politics love can defeat th...   \n",
              "29                  2.0  one see the fake calculation calsi chek kariye...   \n",
              "30                  2.0  being born religion where female deities worsh...   \n",
              "31                  3.0  narendra modi became and despite biased viciou...   \n",
              "32                  6.0  think hindus should back off and let them suff...   \n",
              "33                  3.0  from the very beginningmodi doing wada faramos...   \n",
              "34                  5.0  women are powerful nature wat hinduism states ...   \n",
              "35                  1.0  almost 4000 crore spent but the ganga more pol...   \n",
              "36                  1.0  seriously must have sick mind call small kid r...   \n",
              "37                  7.0  agree with you was unrequired was kinda uncomf...   \n",
              "38                  4.0  rajdeep think 4got that imran not indian never...   \n",
              "39                  6.0  the great modi trap ways congress has walked i...   \n",
              "40                  2.0  has come new meanings nationalism hindu and su...   \n",
              "41                  3.0               modi govt hindus are behaving wildly   \n",
              "42                  1.0  sir one request why bjp candidate contesting f...   \n",
              "43                  1.0  vapas lana hai\\ndesh agey badana hai\\nvote for...   \n",
              "44                  2.0  even with massive mandate 336 seats the nda go...   \n",
              "45                  3.0  can promise what can delivered epf pension uni...   \n",
              "46                  2.0  and even print this seriously whats this elect...   \n",
              "47                  1.0  she has asked three questions from modi and he...   \n",
              "48                  2.0  dear all tsunami favour modi 2019 coming from ...   \n",
              "49                  4.0  rahul gandhis politics love can defeat the mod...   \n",
              "\n",
              "           y  \n",
              "0   positive  \n",
              "1   positive  \n",
              "2   negative  \n",
              "3   positive  \n",
              "4   positive  \n",
              "5   positive  \n",
              "6   negative  \n",
              "7   negative  \n",
              "8   positive  \n",
              "9   positive  \n",
              "10  positive  \n",
              "11  negative  \n",
              "12  positive  \n",
              "13  positive  \n",
              "14  positive  \n",
              "15  positive  \n",
              "16  negative  \n",
              "17  positive  \n",
              "18  positive  \n",
              "19  negative  \n",
              "20  positive  \n",
              "21  positive  \n",
              "22  negative  \n",
              "23  positive  \n",
              "24  negative  \n",
              "25  positive  \n",
              "26  positive  \n",
              "27  negative  \n",
              "28  negative  \n",
              "29  negative  \n",
              "30  positive  \n",
              "31  negative  \n",
              "32  positive  \n",
              "33  negative  \n",
              "34  positive  \n",
              "35  positive  \n",
              "36  negative  \n",
              "37  positive  \n",
              "38  negative  \n",
              "39  positive  \n",
              "40  positive  \n",
              "41  positive  \n",
              "42  negative  \n",
              "43  positive  \n",
              "44  positive  \n",
              "45  positive  \n",
              "46  negative  \n",
              "47  positive  \n",
              "48  negative  \n",
              "49  negative  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d1a90cd4-5a4f-449a-bb7c-10175fa0e6a9\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>document</th>\n",
              "      <th>sentence_embedding_small_bert_L2_128</th>\n",
              "      <th>sentiment</th>\n",
              "      <th>sentiment_confidence</th>\n",
              "      <th>text</th>\n",
              "      <th>y</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>new post added mumbai press official site prod...</td>\n",
              "      <td>[-0.2672112286090851, 0.22553667426109314, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>8.0</td>\n",
              "      <td>new post added mumbai press official site prod...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>not wrong the actual temperature might but and...</td>\n",
              "      <td>[-1.243513822555542, 0.24190887808799744, -0.4...</td>\n",
              "      <td>positive</td>\n",
              "      <td>8.0</td>\n",
              "      <td>not wrong the actual temperature might but and...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>why pakistan crying name modi every day how na...</td>\n",
              "      <td>[-0.7444853782653809, -0.0514342300593853, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>why pakistan crying name modi every day how na...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>congress years wasnt able complete one rafale ...</td>\n",
              "      <td>[-0.34242647886276245, 0.46881920099258423, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>congress years wasnt able complete one rafale ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>public toilet near kanagadurga temple nizampet...</td>\n",
              "      <td>[-1.1381851434707642, 0.512217104434967, -0.74...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>public toilet near kanagadurga temple nizampet...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>the foundation for new india 2022 has already ...</td>\n",
              "      <td>[-0.4057338237762451, 1.0029019117355347, -0.9...</td>\n",
              "      <td>positive</td>\n",
              "      <td>7.0</td>\n",
              "      <td>\\nthe foundation for new india 2022 has alread...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>dear governorani can you let the people indian...</td>\n",
              "      <td>[-1.4633989334106445, 0.0006002967129461467, -...</td>\n",
              "      <td>positive</td>\n",
              "      <td>3.0</td>\n",
              "      <td>dear governorani can you let the people indian...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>this daft donkey’ dick aap was born the iac mo...</td>\n",
              "      <td>[0.04606145992875099, 0.3098487854003906, 0.02...</td>\n",
              "      <td>positive</td>\n",
              "      <td>6.0</td>\n",
              "      <td>this daft donkey’ dick aap was born the iac mo...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>major reason for social hatred and strife modi...</td>\n",
              "      <td>[-0.9470604658126831, 0.27183642983436584, -1....</td>\n",
              "      <td>positive</td>\n",
              "      <td>5.0</td>\n",
              "      <td>major reason for social hatred and strife modi...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>demo was black money caught modi did inspite r...</td>\n",
              "      <td>[-0.7136786580085754, 0.0788763239979744, -0.5...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>demo was black money caught modi did inspite r...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>one the best ministers modis cabinet</td>\n",
              "      <td>[-0.3664279282093048, 0.3727397918701172, -0.4...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>one the best ministers modis cabinet</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>raghuram rajan sent list high profile bank fra...</td>\n",
              "      <td>[-1.0506610870361328, 0.20963071286678314, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>8.0</td>\n",
              "      <td>raghuram rajan sent list high profile bank fra...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>governor kalyan singh aligarh 23rd march all a...</td>\n",
              "      <td>[-0.20803016424179077, 0.07477151602506638, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>governor kalyan singh aligarh 23rd march all a...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>this campaign low hanging fruit seems giving f...</td>\n",
              "      <td>[-1.2681258916854858, 0.24513696134090424, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>3.0</td>\n",
              "      <td>this campaign low hanging fruit seems giving f...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>strict policing that too health system sir reg...</td>\n",
              "      <td>[-0.9757605195045471, 1.0792640447616577, -0.4...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>strict policing that too health system sir reg...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>shatrughan sinha was far bigger public figure ...</td>\n",
              "      <td>[-0.5276148319244385, 0.2546652853488922, -0.1...</td>\n",
              "      <td>positive</td>\n",
              "      <td>6.0</td>\n",
              "      <td>shatrughan sinha was far bigger public figure ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>people wish your vision india and least intere...</td>\n",
              "      <td>[-1.0698672533035278, 0.5003032088279724, -0.4...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>people wish your vision india and least intere...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>chowkidar hee chor hain baap chor beta bada ch...</td>\n",
              "      <td>[-0.5458223223686218, -0.23064681887626648, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>7.0</td>\n",
              "      <td>chowkidar hee chor hain baap chor beta bada ch...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>with modi all his drawbacks atleast know what ...</td>\n",
              "      <td>[-0.8066104054450989, -0.014454682357609272, -...</td>\n",
              "      <td>positive</td>\n",
              "      <td>7.0</td>\n",
              "      <td>with modi all his drawbacks atleast know what ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>compare that with modi’ 2014 “vikas purush” el...</td>\n",
              "      <td>[-0.5401442646980286, -0.41557249426841736, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>compare that with modi’ 2014 “vikas purush” el...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>with welfare delivery gst ibc and feo place mo...</td>\n",
              "      <td>[-0.5764278769493103, 1.2036645412445068, -1.0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>7.0</td>\n",
              "      <td>with welfare delivery gst ibc and feo place mo...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>young and dynamic chowkidar says you are not w...</td>\n",
              "      <td>[-0.08391488343477249, -0.43613195419311523, 0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>young and dynamic chowkidar says you are not w...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>dont forget petrol prices have risen ₹ modi go...</td>\n",
              "      <td>[-0.10981855541467667, 0.5448974370956421, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>5.0</td>\n",
              "      <td>dont forget petrol prices have risen ₹ modi go...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>prime minister narendra modi has urged voters ...</td>\n",
              "      <td>[0.27992355823516846, 0.0582934208214283, -0.2...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>prime minister narendra modi has urged voters ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>when someone asks random question economy sche...</td>\n",
              "      <td>[-0.5603336691856384, -0.1663953959941864, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>when someone asks random question economy sche...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>from whatsapp have two options this elections ...</td>\n",
              "      <td>[0.02258816733956337, -0.18138407170772552, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>from whatsapp have two options this elections ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>heres interesting section work which gives com...</td>\n",
              "      <td>[-1.3662021160125732, 0.042950600385665894, 0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>heres interesting section work which gives com...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>know why you willpapu his chamchas nothing tar...</td>\n",
              "      <td>[-1.0259658098220825, 0.31534343957901, -0.242...</td>\n",
              "      <td>positive</td>\n",
              "      <td>6.0</td>\n",
              "      <td>know why you willpapu his chamchas nothing tar...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>only rahul gandhis politics love can defeat th...</td>\n",
              "      <td>[-0.411965548992157, -0.5224093198776245, -0.6...</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>only rahul gandhis politics love can defeat th...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>one see the fake calculation calsi chek kariye...</td>\n",
              "      <td>[-1.0891247987747192, -0.5220181345939636, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>one see the fake calculation calsi chek kariye...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>being born religion where female deities worsh...</td>\n",
              "      <td>[-1.0681140422821045, -0.6835974454879761, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>being born religion where female deities worsh...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>narendra modi became and despite biased viciou...</td>\n",
              "      <td>[-0.20442193746566772, -0.3683846890926361, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>3.0</td>\n",
              "      <td>narendra modi became and despite biased viciou...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>think hindus should back off and let them suff...</td>\n",
              "      <td>[-0.9112239480018616, 0.17845268547534943, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>6.0</td>\n",
              "      <td>think hindus should back off and let them suff...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>from the very beginningmodi doing wada faramos...</td>\n",
              "      <td>[-1.3499250411987305, 0.23698307573795319, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>3.0</td>\n",
              "      <td>from the very beginningmodi doing wada faramos...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>women are powerful nature wat hinduism states ...</td>\n",
              "      <td>[-0.47838863730430603, 0.19593238830566406, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>5.0</td>\n",
              "      <td>women are powerful nature wat hinduism states ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>almost 4000 crore spent but the ganga more pol...</td>\n",
              "      <td>[-1.229308009147644, -0.07380063086748123, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>almost 4000 crore spent but the ganga more pol...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>seriously must have sick mind call small kid r...</td>\n",
              "      <td>[-0.8348453640937805, -0.03178590536117554, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>seriously must have sick mind call small kid r...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>agree with you was unrequired was kinda uncomf...</td>\n",
              "      <td>[-0.4827183485031128, 0.08657485246658325, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>7.0</td>\n",
              "      <td>agree with you was unrequired was kinda uncomf...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>rajdeep think 4got that imran not indian never...</td>\n",
              "      <td>[-0.6942201852798462, -0.3642423152923584, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>4.0</td>\n",
              "      <td>rajdeep think 4got that imran not indian never...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>the great modi trap ways congress has walked into</td>\n",
              "      <td>[-0.9660191535949707, -0.2219739854335785, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>6.0</td>\n",
              "      <td>the great modi trap ways congress has walked i...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>has come new meanings nationalism hindu and su...</td>\n",
              "      <td>[-0.03318127244710922, 0.04329724237322807, 0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>has come new meanings nationalism hindu and su...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>modi govt hindus are behaving wildly</td>\n",
              "      <td>[-0.5563615560531616, 0.4725855588912964, 0.10...</td>\n",
              "      <td>positive</td>\n",
              "      <td>3.0</td>\n",
              "      <td>modi govt hindus are behaving wildly</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>sir one request why bjp candidate contesting f...</td>\n",
              "      <td>[-0.6955257654190063, 0.37047961354255676, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>sir one request why bjp candidate contesting f...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>vapas lana hai desh agey badana hai vote for m...</td>\n",
              "      <td>[-0.8704789876937866, -0.22370557487010956, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>vapas lana hai\\ndesh agey badana hai\\nvote for...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>even with massive mandate 336 seats the nda go...</td>\n",
              "      <td>[-0.20755957067012787, 0.34972018003463745, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>even with massive mandate 336 seats the nda go...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>can promise what can delivered epf pension uni...</td>\n",
              "      <td>[-1.3415300846099854, 1.6326956748962402, -0.6...</td>\n",
              "      <td>positive</td>\n",
              "      <td>3.0</td>\n",
              "      <td>can promise what can delivered epf pension uni...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>and even print this seriously whats this elect...</td>\n",
              "      <td>[-1.4377732276916504, -0.11478081345558167, 0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>and even print this seriously whats this elect...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>she has asked three questions from modi and he...</td>\n",
              "      <td>[-1.193617343902588, -0.02149897627532482, 0.1...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>she has asked three questions from modi and he...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>dear all tsunami favour modi 2019 coming from ...</td>\n",
              "      <td>[-0.5854026675224304, -0.21378959715366364, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>dear all tsunami favour modi 2019 coming from ...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>rahul gandhis politics love can defeat the mod...</td>\n",
              "      <td>[-0.39933672547340393, -0.5969381332397461, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>4.0</td>\n",
              "      <td>rahul gandhis politics love can defeat the mod...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d1a90cd4-5a4f-449a-bb7c-10175fa0e6a9')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-d1a90cd4-5a4f-449a-bb7c-10175fa0e6a9 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-d1a90cd4-5a4f-449a-bb7c-10175fa0e6a9');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-67d4cb1b-c943-4ec1-86aa-7fc2d878557a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-67d4cb1b-c943-4ec1-86aa-7fc2d878557a')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-67d4cb1b-c943-4ec1-86aa-7fc2d878557a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lVyOE2wV0fw_"
      },
      "source": [
        "# Test the fitted pipe on new example"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 150
        },
        "id": "qdCUg2MR0PD2",
        "outputId": "7372f9d4-4492-4784-a7dd-d1bef74201bb"
      },
      "source": [
        "fitted_pipe.predict('the president of india just died')"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "sentence_detector_dl download started this may take some time.\n",
            "Approximate size to download 354.6 KB\n",
            "[OK!]\n",
            "Warning::Spark Session already created, some configs may not take.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                           sentence  \\\n",
              "0  the president of india just died   \n",
              "\n",
              "                sentence_embedding_small_bert_L2_128 sentiment  \\\n",
              "0  [-0.9852966070175171, 0.5659735798835754, -1.0...  positive   \n",
              "\n",
              "  sentiment_confidence  \n",
              "0                  1.0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f249f3ae-2d4a-48da-8cbf-97b3d6a85555\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>sentence_embedding_small_bert_L2_128</th>\n",
              "      <th>sentiment</th>\n",
              "      <th>sentiment_confidence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>the president of india just died</td>\n",
              "      <td>[-0.9852966070175171, 0.5659735798835754, -1.0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f249f3ae-2d4a-48da-8cbf-97b3d6a85555')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-f249f3ae-2d4a-48da-8cbf-97b3d6a85555 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-f249f3ae-2d4a-48da-8cbf-97b3d6a85555');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xflpwrVjjBVD"
      },
      "source": [
        "## Configure pipe training parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UtsAUGTmOTms",
        "outputId": "e3503721-ee9a-4c98-b35b-d861cbc98880"
      },
      "source": [
        "trainable_pipe.print_info()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The following parameters are configurable for this NLU pipeline (You can copy paste the examples) :\n",
            ">>> component_list['bert_sentence_embeddings@sent_small_bert_L2_128'] has settable params:\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L2_128'].setBatchSize(8)              | Info: Size of every batch | Currently set to : 8\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L2_128'].setEngine('tensorflow')      | Info: Deep Learning engine used for this model | Currently set to : tensorflow\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L2_128'].setIsLong(False)             | Info: Use Long type instead of Int type for inputs buffer - Some Bert models require Long instead of Int. | Currently set to : False\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L2_128'].setMaxSentenceLength(128)    | Info: Max sentence length to process | Currently set to : 128\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L2_128'].setDimension(128)            | Info: Number of embedding dimensions | Currently set to : 128\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L2_128'].setCaseSensitive(False)      | Info: whether to ignore case in tokens for embeddings matching | Currently set to : False\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L2_128'].setStorageRef('sent_small_bert_L2_128')  | Info: unique reference name for identification | Currently set to : sent_small_bert_L2_128\n",
            ">>> component_list['document_assembler'] has settable params:\n",
            "component_list['document_assembler'].setCleanupMode('shrink')                                  | Info: possible values: disabled, inplace, inplace_full, shrink, shrink_full, each, each_full, delete_full | Currently set to : shrink\n",
            ">>> component_list['sentiment_dl@sent_small_bert_L2_128'] has settable params:\n",
            "component_list['sentiment_dl@sent_small_bert_L2_128'].setEngine('tensorflow')                  | Info: Deep Learning engine used for this model | Currently set to : tensorflow\n",
            "component_list['sentiment_dl@sent_small_bert_L2_128'].setThreshold(0.6)                        | Info: The minimum threshold for the final result otheriwse it will be neutral | Currently set to : 0.6\n",
            "component_list['sentiment_dl@sent_small_bert_L2_128'].setThresholdLabel('neutral')             | Info: In case the score is less than threshold, what should be the label. Default is neutral. | Currently set to : neutral\n",
            "component_list['sentiment_dl@sent_small_bert_L2_128'].setStorageRef('sent_small_bert_L2_128')  | Info: unique reference name for identification | Currently set to : sent_small_bert_L2_128\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2GJdDNV9jEIe"
      },
      "source": [
        "## Retrain with new parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "mptfvHx-MMMX",
        "outputId": "ffccac34-9684-41f4-d1e7-7f07951fd516"
      },
      "source": [
        "# Train longer!\n",
        "trainable_pipe = nlp.load('train.sentiment')\n",
        "trainable_pipe['trainable_sentiment_dl'].setMaxEpochs(5)\n",
        "fitted_pipe = trainable_pipe.fit(train_df.iloc[:50])\n",
        "# predict with the trainable pipeline on dataset and get predictions\n",
        "preds = fitted_pipe.predict(train_df.iloc[:50],output_level='document')\n",
        "\n",
        "#sentence detector that is part of the pipe generates sone NaNs. lets drop them first\n",
        "preds.dropna(inplace=True)\n",
        "print(classification_report(preds['y'], preds['sentiment']))\n",
        "\n",
        "preds"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Warning::Spark Session already created, some configs may not take.\n",
            "Warning::Spark Session already created, some configs may not take.\n",
            "sent_small_bert_L2_128 download started this may take some time.\n",
            "Approximate size to download 16.1 MB\n",
            "[OK!]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/nlu/pipe/pipeline.py:149: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  dataset.y = dataset.y.apply(str)\n",
            "/usr/local/lib/python3.10/dist-packages/nlu/pipe/utils/data_conversion_utils.py:160: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  data['origin_index'] = data.index\n",
            "/usr/local/lib/python3.10/dist-packages/nlu/pipe/utils/data_conversion_utils.py:160: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  data['origin_index'] = data.index\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "    negative       0.00      0.00      0.00        19\n",
            "    positive       0.62      1.00      0.77        31\n",
            "\n",
            "    accuracy                           0.62        50\n",
            "   macro avg       0.31      0.50      0.38        50\n",
            "weighted avg       0.38      0.62      0.47        50\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/nlu/pipe/extractors/extractor_methods/base_extractor_methods.py:356: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  df[cols_to_explode] = df[cols_to_explode].apply(pad_same_level_cols, axis=1)\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                             document  \\\n",
              "0   new post added mumbai press official site prod...   \n",
              "1   not wrong the actual temperature might but and...   \n",
              "2   why pakistan crying name modi every day how na...   \n",
              "3   congress years wasnt able complete one rafale ...   \n",
              "4   public toilet near kanagadurga temple nizampet...   \n",
              "5   the foundation for new india 2022 has already ...   \n",
              "6   dear governorani can you let the people indian...   \n",
              "7   this daft donkey’ dick aap was born the iac mo...   \n",
              "8   major reason for social hatred and strife modi...   \n",
              "9   demo was black money caught modi did inspite r...   \n",
              "10               one the best ministers modis cabinet   \n",
              "11  raghuram rajan sent list high profile bank fra...   \n",
              "12  governor kalyan singh aligarh 23rd march all a...   \n",
              "13  this campaign low hanging fruit seems giving f...   \n",
              "14  strict policing that too health system sir reg...   \n",
              "15  shatrughan sinha was far bigger public figure ...   \n",
              "16  people wish your vision india and least intere...   \n",
              "17  chowkidar hee chor hain baap chor beta bada ch...   \n",
              "18  with modi all his drawbacks atleast know what ...   \n",
              "19  compare that with modi’ 2014 “vikas purush” el...   \n",
              "20  with welfare delivery gst ibc and feo place mo...   \n",
              "21  young and dynamic chowkidar says you are not w...   \n",
              "22  dont forget petrol prices have risen ₹ modi go...   \n",
              "23  prime minister narendra modi has urged voters ...   \n",
              "24  when someone asks random question economy sche...   \n",
              "25  from whatsapp have two options this elections ...   \n",
              "26  heres interesting section work which gives com...   \n",
              "27  know why you willpapu his chamchas nothing tar...   \n",
              "28  only rahul gandhis politics love can defeat th...   \n",
              "29  one see the fake calculation calsi chek kariye...   \n",
              "30  being born religion where female deities worsh...   \n",
              "31  narendra modi became and despite biased viciou...   \n",
              "32  think hindus should back off and let them suff...   \n",
              "33  from the very beginningmodi doing wada faramos...   \n",
              "34  women are powerful nature wat hinduism states ...   \n",
              "35  almost 4000 crore spent but the ganga more pol...   \n",
              "36  seriously must have sick mind call small kid r...   \n",
              "37  agree with you was unrequired was kinda uncomf...   \n",
              "38  rajdeep think 4got that imran not indian never...   \n",
              "39  the great modi trap ways congress has walked into   \n",
              "40  has come new meanings nationalism hindu and su...   \n",
              "41               modi govt hindus are behaving wildly   \n",
              "42  sir one request why bjp candidate contesting f...   \n",
              "43  vapas lana hai desh agey badana hai vote for m...   \n",
              "44  even with massive mandate 336 seats the nda go...   \n",
              "45  can promise what can delivered epf pension uni...   \n",
              "46  and even print this seriously whats this elect...   \n",
              "47  she has asked three questions from modi and he...   \n",
              "48  dear all tsunami favour modi 2019 coming from ...   \n",
              "49  rahul gandhis politics love can defeat the mod...   \n",
              "\n",
              "                 sentence_embedding_small_bert_L2_128 sentiment  \\\n",
              "0   [-0.2672112286090851, 0.22553667426109314, -0....  positive   \n",
              "1   [-1.243513822555542, 0.24190887808799744, -0.4...  positive   \n",
              "2   [-0.7444853782653809, -0.0514342300593853, -0....  positive   \n",
              "3   [-0.34242647886276245, 0.46881920099258423, -0...  positive   \n",
              "4   [-1.1381851434707642, 0.512217104434967, -0.74...  positive   \n",
              "5   [-0.4057338237762451, 1.0029019117355347, -0.9...  positive   \n",
              "6   [-1.4633989334106445, 0.0006002967129461467, -...  positive   \n",
              "7   [0.04606145992875099, 0.3098487854003906, 0.02...  positive   \n",
              "8   [-0.9470604658126831, 0.27183642983436584, -1....  positive   \n",
              "9   [-0.7136786580085754, 0.0788763239979744, -0.5...  positive   \n",
              "10  [-0.3664279282093048, 0.3727397918701172, -0.4...  positive   \n",
              "11  [-1.0506610870361328, 0.20963071286678314, -0....  positive   \n",
              "12  [-0.20803016424179077, 0.07477151602506638, -0...  positive   \n",
              "13  [-1.2681258916854858, 0.24513696134090424, -0....  positive   \n",
              "14  [-0.9757605195045471, 1.0792640447616577, -0.4...  positive   \n",
              "15  [-0.5276148319244385, 0.2546652853488922, -0.1...  positive   \n",
              "16  [-1.0698672533035278, 0.5003032088279724, -0.4...  positive   \n",
              "17  [-0.5458223223686218, -0.23064681887626648, -0...  positive   \n",
              "18  [-0.8066104054450989, -0.014454682357609272, -...  positive   \n",
              "19  [-0.5401442646980286, -0.41557249426841736, -0...  positive   \n",
              "20  [-0.5764278769493103, 1.2036645412445068, -1.0...  positive   \n",
              "21  [-0.08391488343477249, -0.43613195419311523, 0...  positive   \n",
              "22  [-0.10981855541467667, 0.5448974370956421, -0....  positive   \n",
              "23  [0.27992355823516846, 0.0582934208214283, -0.2...  positive   \n",
              "24  [-0.5603336691856384, -0.1663953959941864, -0....  positive   \n",
              "25  [0.02258816733956337, -0.18138407170772552, -0...  positive   \n",
              "26  [-1.3662021160125732, 0.042950600385665894, 0....  positive   \n",
              "27  [-1.0259658098220825, 0.31534343957901, -0.242...  positive   \n",
              "28  [-0.411965548992157, -0.5224093198776245, -0.6...  positive   \n",
              "29  [-1.0891247987747192, -0.5220181345939636, -0....  positive   \n",
              "30  [-1.0681140422821045, -0.6835974454879761, -0....  positive   \n",
              "31  [-0.20442193746566772, -0.3683846890926361, -0...  positive   \n",
              "32  [-0.9112239480018616, 0.17845268547534943, -0....  positive   \n",
              "33  [-1.3499250411987305, 0.23698307573795319, -0....  positive   \n",
              "34  [-0.47838863730430603, 0.19593238830566406, -0...  positive   \n",
              "35  [-1.229308009147644, -0.07380063086748123, -0....  positive   \n",
              "36  [-0.8348453640937805, -0.03178590536117554, -0...  positive   \n",
              "37  [-0.4827183485031128, 0.08657485246658325, -0....  positive   \n",
              "38  [-0.6942201852798462, -0.3642423152923584, -0....  positive   \n",
              "39  [-0.9660191535949707, -0.2219739854335785, -0....  positive   \n",
              "40  [-0.03318127244710922, 0.04329724237322807, 0....  positive   \n",
              "41  [-0.5563615560531616, 0.4725855588912964, 0.10...  positive   \n",
              "42  [-0.6955257654190063, 0.37047961354255676, -0....  positive   \n",
              "43  [-0.8704789876937866, -0.22370557487010956, -0...  positive   \n",
              "44  [-0.20755957067012787, 0.34972018003463745, -0...  positive   \n",
              "45  [-1.3415300846099854, 1.6326956748962402, -0.6...  positive   \n",
              "46  [-1.4377732276916504, -0.11478081345558167, 0....  positive   \n",
              "47  [-1.193617343902588, -0.02149897627532482, 0.1...  positive   \n",
              "48  [-0.5854026675224304, -0.21378959715366364, -0...  positive   \n",
              "49  [-0.39933672547340393, -0.5969381332397461, -0...  positive   \n",
              "\n",
              "   sentiment_confidence                                               text  \\\n",
              "0                   5.0  new post added mumbai press official site prod...   \n",
              "1                   1.0  not wrong the actual temperature might but and...   \n",
              "2                   1.0  why pakistan crying name modi every day how na...   \n",
              "3                   8.0  congress years wasnt able complete one rafale ...   \n",
              "4                   2.0  public toilet near kanagadurga temple nizampet...   \n",
              "5                   9.0  \\nthe foundation for new india 2022 has alread...   \n",
              "6                   2.0  dear governorani can you let the people indian...   \n",
              "7                   1.0  this daft donkey’ dick aap was born the iac mo...   \n",
              "8                   1.0  major reason for social hatred and strife modi...   \n",
              "9                   1.0  demo was black money caught modi did inspite r...   \n",
              "10                  5.0              one the best ministers modis cabinet    \n",
              "11                  3.0  raghuram rajan sent list high profile bank fra...   \n",
              "12                  8.0  governor kalyan singh aligarh 23rd march all a...   \n",
              "13                  2.0  this campaign low hanging fruit seems giving f...   \n",
              "14                  2.0  strict policing that too health system sir reg...   \n",
              "15                  8.0  shatrughan sinha was far bigger public figure ...   \n",
              "16                  3.0  people wish your vision india and least intere...   \n",
              "17                  4.0  chowkidar hee chor hain baap chor beta bada ch...   \n",
              "18                  1.0  with modi all his drawbacks atleast know what ...   \n",
              "19                  6.0  compare that with modi’ 2014 “vikas purush” el...   \n",
              "20                  1.0  with welfare delivery gst ibc and feo place mo...   \n",
              "21                  5.0  young and dynamic chowkidar says you are not w...   \n",
              "22                  1.0  dont forget petrol prices have risen ₹ modi go...   \n",
              "23                  7.0  prime minister narendra modi has urged voters ...   \n",
              "24                  2.0  when someone asks random question economy sche...   \n",
              "25                  5.0  from whatsapp have two options this elections ...   \n",
              "26                  1.0  heres interesting section work which gives com...   \n",
              "27                  1.0  know why you willpapu his chamchas nothing tar...   \n",
              "28                  6.0  only rahul gandhis politics love can defeat th...   \n",
              "29                  6.0  one see the fake calculation calsi chek kariye...   \n",
              "30                  2.0  being born religion where female deities worsh...   \n",
              "31                  1.0  narendra modi became and despite biased viciou...   \n",
              "32                  2.0  think hindus should back off and let them suff...   \n",
              "33                  2.0  from the very beginningmodi doing wada faramos...   \n",
              "34                  1.0  women are powerful nature wat hinduism states ...   \n",
              "35                  2.0  almost 4000 crore spent but the ganga more pol...   \n",
              "36                  4.0  seriously must have sick mind call small kid r...   \n",
              "37                  2.0  agree with you was unrequired was kinda uncomf...   \n",
              "38                  2.0  rajdeep think 4got that imran not indian never...   \n",
              "39                  9.0  the great modi trap ways congress has walked i...   \n",
              "40                  1.0  has come new meanings nationalism hindu and su...   \n",
              "41                  1.0               modi govt hindus are behaving wildly   \n",
              "42                  1.0  sir one request why bjp candidate contesting f...   \n",
              "43                  1.0  vapas lana hai\\ndesh agey badana hai\\nvote for...   \n",
              "44                  4.0  even with massive mandate 336 seats the nda go...   \n",
              "45                  2.0  can promise what can delivered epf pension uni...   \n",
              "46                  9.0  and even print this seriously whats this elect...   \n",
              "47                  1.0  she has asked three questions from modi and he...   \n",
              "48                  4.0  dear all tsunami favour modi 2019 coming from ...   \n",
              "49                  5.0  rahul gandhis politics love can defeat the mod...   \n",
              "\n",
              "           y  \n",
              "0   positive  \n",
              "1   positive  \n",
              "2   negative  \n",
              "3   positive  \n",
              "4   positive  \n",
              "5   positive  \n",
              "6   negative  \n",
              "7   negative  \n",
              "8   positive  \n",
              "9   positive  \n",
              "10  positive  \n",
              "11  negative  \n",
              "12  positive  \n",
              "13  positive  \n",
              "14  positive  \n",
              "15  positive  \n",
              "16  negative  \n",
              "17  positive  \n",
              "18  positive  \n",
              "19  negative  \n",
              "20  positive  \n",
              "21  positive  \n",
              "22  negative  \n",
              "23  positive  \n",
              "24  negative  \n",
              "25  positive  \n",
              "26  positive  \n",
              "27  negative  \n",
              "28  negative  \n",
              "29  negative  \n",
              "30  positive  \n",
              "31  negative  \n",
              "32  positive  \n",
              "33  negative  \n",
              "34  positive  \n",
              "35  positive  \n",
              "36  negative  \n",
              "37  positive  \n",
              "38  negative  \n",
              "39  positive  \n",
              "40  positive  \n",
              "41  positive  \n",
              "42  negative  \n",
              "43  positive  \n",
              "44  positive  \n",
              "45  positive  \n",
              "46  negative  \n",
              "47  positive  \n",
              "48  negative  \n",
              "49  negative  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f72f8034-8b8a-4a6c-858c-2e6347a0f93b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>document</th>\n",
              "      <th>sentence_embedding_small_bert_L2_128</th>\n",
              "      <th>sentiment</th>\n",
              "      <th>sentiment_confidence</th>\n",
              "      <th>text</th>\n",
              "      <th>y</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>new post added mumbai press official site prod...</td>\n",
              "      <td>[-0.2672112286090851, 0.22553667426109314, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>5.0</td>\n",
              "      <td>new post added mumbai press official site prod...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>not wrong the actual temperature might but and...</td>\n",
              "      <td>[-1.243513822555542, 0.24190887808799744, -0.4...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>not wrong the actual temperature might but and...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>why pakistan crying name modi every day how na...</td>\n",
              "      <td>[-0.7444853782653809, -0.0514342300593853, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>why pakistan crying name modi every day how na...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>congress years wasnt able complete one rafale ...</td>\n",
              "      <td>[-0.34242647886276245, 0.46881920099258423, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>8.0</td>\n",
              "      <td>congress years wasnt able complete one rafale ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>public toilet near kanagadurga temple nizampet...</td>\n",
              "      <td>[-1.1381851434707642, 0.512217104434967, -0.74...</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>public toilet near kanagadurga temple nizampet...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>the foundation for new india 2022 has already ...</td>\n",
              "      <td>[-0.4057338237762451, 1.0029019117355347, -0.9...</td>\n",
              "      <td>positive</td>\n",
              "      <td>9.0</td>\n",
              "      <td>\\nthe foundation for new india 2022 has alread...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>dear governorani can you let the people indian...</td>\n",
              "      <td>[-1.4633989334106445, 0.0006002967129461467, -...</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>dear governorani can you let the people indian...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>this daft donkey’ dick aap was born the iac mo...</td>\n",
              "      <td>[0.04606145992875099, 0.3098487854003906, 0.02...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>this daft donkey’ dick aap was born the iac mo...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>major reason for social hatred and strife modi...</td>\n",
              "      <td>[-0.9470604658126831, 0.27183642983436584, -1....</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>major reason for social hatred and strife modi...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>demo was black money caught modi did inspite r...</td>\n",
              "      <td>[-0.7136786580085754, 0.0788763239979744, -0.5...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>demo was black money caught modi did inspite r...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>one the best ministers modis cabinet</td>\n",
              "      <td>[-0.3664279282093048, 0.3727397918701172, -0.4...</td>\n",
              "      <td>positive</td>\n",
              "      <td>5.0</td>\n",
              "      <td>one the best ministers modis cabinet</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>raghuram rajan sent list high profile bank fra...</td>\n",
              "      <td>[-1.0506610870361328, 0.20963071286678314, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>3.0</td>\n",
              "      <td>raghuram rajan sent list high profile bank fra...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>governor kalyan singh aligarh 23rd march all a...</td>\n",
              "      <td>[-0.20803016424179077, 0.07477151602506638, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>8.0</td>\n",
              "      <td>governor kalyan singh aligarh 23rd march all a...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>this campaign low hanging fruit seems giving f...</td>\n",
              "      <td>[-1.2681258916854858, 0.24513696134090424, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>this campaign low hanging fruit seems giving f...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>strict policing that too health system sir reg...</td>\n",
              "      <td>[-0.9757605195045471, 1.0792640447616577, -0.4...</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>strict policing that too health system sir reg...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>shatrughan sinha was far bigger public figure ...</td>\n",
              "      <td>[-0.5276148319244385, 0.2546652853488922, -0.1...</td>\n",
              "      <td>positive</td>\n",
              "      <td>8.0</td>\n",
              "      <td>shatrughan sinha was far bigger public figure ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>people wish your vision india and least intere...</td>\n",
              "      <td>[-1.0698672533035278, 0.5003032088279724, -0.4...</td>\n",
              "      <td>positive</td>\n",
              "      <td>3.0</td>\n",
              "      <td>people wish your vision india and least intere...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>chowkidar hee chor hain baap chor beta bada ch...</td>\n",
              "      <td>[-0.5458223223686218, -0.23064681887626648, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>4.0</td>\n",
              "      <td>chowkidar hee chor hain baap chor beta bada ch...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>with modi all his drawbacks atleast know what ...</td>\n",
              "      <td>[-0.8066104054450989, -0.014454682357609272, -...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>with modi all his drawbacks atleast know what ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>compare that with modi’ 2014 “vikas purush” el...</td>\n",
              "      <td>[-0.5401442646980286, -0.41557249426841736, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>6.0</td>\n",
              "      <td>compare that with modi’ 2014 “vikas purush” el...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>with welfare delivery gst ibc and feo place mo...</td>\n",
              "      <td>[-0.5764278769493103, 1.2036645412445068, -1.0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>with welfare delivery gst ibc and feo place mo...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>young and dynamic chowkidar says you are not w...</td>\n",
              "      <td>[-0.08391488343477249, -0.43613195419311523, 0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>5.0</td>\n",
              "      <td>young and dynamic chowkidar says you are not w...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>dont forget petrol prices have risen ₹ modi go...</td>\n",
              "      <td>[-0.10981855541467667, 0.5448974370956421, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>dont forget petrol prices have risen ₹ modi go...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>prime minister narendra modi has urged voters ...</td>\n",
              "      <td>[0.27992355823516846, 0.0582934208214283, -0.2...</td>\n",
              "      <td>positive</td>\n",
              "      <td>7.0</td>\n",
              "      <td>prime minister narendra modi has urged voters ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>when someone asks random question economy sche...</td>\n",
              "      <td>[-0.5603336691856384, -0.1663953959941864, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>when someone asks random question economy sche...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>from whatsapp have two options this elections ...</td>\n",
              "      <td>[0.02258816733956337, -0.18138407170772552, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>5.0</td>\n",
              "      <td>from whatsapp have two options this elections ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>heres interesting section work which gives com...</td>\n",
              "      <td>[-1.3662021160125732, 0.042950600385665894, 0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>heres interesting section work which gives com...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>know why you willpapu his chamchas nothing tar...</td>\n",
              "      <td>[-1.0259658098220825, 0.31534343957901, -0.242...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>know why you willpapu his chamchas nothing tar...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>only rahul gandhis politics love can defeat th...</td>\n",
              "      <td>[-0.411965548992157, -0.5224093198776245, -0.6...</td>\n",
              "      <td>positive</td>\n",
              "      <td>6.0</td>\n",
              "      <td>only rahul gandhis politics love can defeat th...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>one see the fake calculation calsi chek kariye...</td>\n",
              "      <td>[-1.0891247987747192, -0.5220181345939636, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>6.0</td>\n",
              "      <td>one see the fake calculation calsi chek kariye...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>being born religion where female deities worsh...</td>\n",
              "      <td>[-1.0681140422821045, -0.6835974454879761, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>being born religion where female deities worsh...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>narendra modi became and despite biased viciou...</td>\n",
              "      <td>[-0.20442193746566772, -0.3683846890926361, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>narendra modi became and despite biased viciou...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>think hindus should back off and let them suff...</td>\n",
              "      <td>[-0.9112239480018616, 0.17845268547534943, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>think hindus should back off and let them suff...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>from the very beginningmodi doing wada faramos...</td>\n",
              "      <td>[-1.3499250411987305, 0.23698307573795319, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>from the very beginningmodi doing wada faramos...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>women are powerful nature wat hinduism states ...</td>\n",
              "      <td>[-0.47838863730430603, 0.19593238830566406, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>women are powerful nature wat hinduism states ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>almost 4000 crore spent but the ganga more pol...</td>\n",
              "      <td>[-1.229308009147644, -0.07380063086748123, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>almost 4000 crore spent but the ganga more pol...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>seriously must have sick mind call small kid r...</td>\n",
              "      <td>[-0.8348453640937805, -0.03178590536117554, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>4.0</td>\n",
              "      <td>seriously must have sick mind call small kid r...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>agree with you was unrequired was kinda uncomf...</td>\n",
              "      <td>[-0.4827183485031128, 0.08657485246658325, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>agree with you was unrequired was kinda uncomf...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>rajdeep think 4got that imran not indian never...</td>\n",
              "      <td>[-0.6942201852798462, -0.3642423152923584, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>rajdeep think 4got that imran not indian never...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>the great modi trap ways congress has walked into</td>\n",
              "      <td>[-0.9660191535949707, -0.2219739854335785, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>9.0</td>\n",
              "      <td>the great modi trap ways congress has walked i...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>has come new meanings nationalism hindu and su...</td>\n",
              "      <td>[-0.03318127244710922, 0.04329724237322807, 0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>has come new meanings nationalism hindu and su...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>modi govt hindus are behaving wildly</td>\n",
              "      <td>[-0.5563615560531616, 0.4725855588912964, 0.10...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>modi govt hindus are behaving wildly</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>sir one request why bjp candidate contesting f...</td>\n",
              "      <td>[-0.6955257654190063, 0.37047961354255676, -0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>sir one request why bjp candidate contesting f...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>vapas lana hai desh agey badana hai vote for m...</td>\n",
              "      <td>[-0.8704789876937866, -0.22370557487010956, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>vapas lana hai\\ndesh agey badana hai\\nvote for...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>even with massive mandate 336 seats the nda go...</td>\n",
              "      <td>[-0.20755957067012787, 0.34972018003463745, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>4.0</td>\n",
              "      <td>even with massive mandate 336 seats the nda go...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>can promise what can delivered epf pension uni...</td>\n",
              "      <td>[-1.3415300846099854, 1.6326956748962402, -0.6...</td>\n",
              "      <td>positive</td>\n",
              "      <td>2.0</td>\n",
              "      <td>can promise what can delivered epf pension uni...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>and even print this seriously whats this elect...</td>\n",
              "      <td>[-1.4377732276916504, -0.11478081345558167, 0....</td>\n",
              "      <td>positive</td>\n",
              "      <td>9.0</td>\n",
              "      <td>and even print this seriously whats this elect...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>she has asked three questions from modi and he...</td>\n",
              "      <td>[-1.193617343902588, -0.02149897627532482, 0.1...</td>\n",
              "      <td>positive</td>\n",
              "      <td>1.0</td>\n",
              "      <td>she has asked three questions from modi and he...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>dear all tsunami favour modi 2019 coming from ...</td>\n",
              "      <td>[-0.5854026675224304, -0.21378959715366364, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>4.0</td>\n",
              "      <td>dear all tsunami favour modi 2019 coming from ...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>rahul gandhis politics love can defeat the mod...</td>\n",
              "      <td>[-0.39933672547340393, -0.5969381332397461, -0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>5.0</td>\n",
              "      <td>rahul gandhis politics love can defeat the mod...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f72f8034-8b8a-4a6c-858c-2e6347a0f93b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-f72f8034-8b8a-4a6c-858c-2e6347a0f93b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-f72f8034-8b8a-4a6c-858c-2e6347a0f93b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-124c98c9-2c4f-4ca1-8c4d-6f76b0041a6d\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-124c98c9-2c4f-4ca1-8c4d-6f76b0041a6d')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-124c98c9-2c4f-4ca1-8c4d-6f76b0041a6d button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qFoT-s1MjTSS"
      },
      "source": [
        "# Try training with different Embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nxWFzQOhjWC8",
        "outputId": "6e781fd6-e1eb-4641-b3e1-6f6f4e50b7a8"
      },
      "source": [
        "# We can use nlu.print_components(action='embed_sentence') to see every possibler sentence embedding we could use. Lets use bert!\n",
        "nlp.nlu.print_components(action='embed_sentence')"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "For language <am> NLU provides the following Models : \n",
            "nlu.load('am.embed_sentence.xlm_roberta') returns Spark NLP model_anno_obj sent_xlm_roberta_base_finetuned_amharic\n",
            "For language <de> NLU provides the following Models : \n",
            "nlu.load('de.embed_sentence.bert.base_cased') returns Spark NLP model_anno_obj sent_bert_base_cased\n",
            "For language <el> NLU provides the following Models : \n",
            "nlu.load('el.embed_sentence.bert.base_uncased') returns Spark NLP model_anno_obj sent_bert_base_uncased\n",
            "For language <en> NLU provides the following Models : \n",
            "nlu.load('en.embed_sentence') returns Spark NLP model_anno_obj tfhub_use\n",
            "nlu.load('en.embed_sentence.albert') returns Spark NLP model_anno_obj albert_base_uncased\n",
            "nlu.load('en.embed_sentence.bert') returns Spark NLP model_anno_obj sent_bert_base_uncased\n",
            "nlu.load('en.embed_sentence.bert.base_uncased_legal') returns Spark NLP model_anno_obj sent_bert_base_uncased_legal\n",
            "nlu.load('en.embed_sentence.bert.finetuned') returns Spark NLP model_anno_obj sbert_setfit_finetuned_financial_text_classification\n",
            "nlu.load('en.embed_sentence.bert.pubmed') returns Spark NLP model_anno_obj sent_bert_pubmed\n",
            "nlu.load('en.embed_sentence.bert.pubmed_squad2') returns Spark NLP model_anno_obj sent_bert_pubmed_squad2\n",
            "nlu.load('en.embed_sentence.bert.wiki_books') returns Spark NLP model_anno_obj sent_bert_wiki_books\n",
            "nlu.load('en.embed_sentence.bert.wiki_books_mnli') returns Spark NLP model_anno_obj sent_bert_wiki_books_mnli\n",
            "nlu.load('en.embed_sentence.bert.wiki_books_qnli') returns Spark NLP model_anno_obj sent_bert_wiki_books_qnli\n",
            "nlu.load('en.embed_sentence.bert.wiki_books_qqp') returns Spark NLP model_anno_obj sent_bert_wiki_books_qqp\n",
            "nlu.load('en.embed_sentence.bert.wiki_books_squad2') returns Spark NLP model_anno_obj sent_bert_wiki_books_squad2\n",
            "nlu.load('en.embed_sentence.bert.wiki_books_sst2') returns Spark NLP model_anno_obj sent_bert_wiki_books_sst2\n",
            "nlu.load('en.embed_sentence.bert_base_cased') returns Spark NLP model_anno_obj sent_bert_base_cased\n",
            "nlu.load('en.embed_sentence.bert_base_uncased') returns Spark NLP model_anno_obj sent_bert_base_uncased\n",
            "nlu.load('en.embed_sentence.bert_large_cased') returns Spark NLP model_anno_obj sent_bert_large_cased\n",
            "nlu.load('en.embed_sentence.bert_large_uncased') returns Spark NLP model_anno_obj sent_bert_large_uncased\n",
            "nlu.load('en.embed_sentence.bert_use_cmlm_en_base') returns Spark NLP model_anno_obj sent_bert_use_cmlm_en_base\n",
            "nlu.load('en.embed_sentence.bert_use_cmlm_en_large') returns Spark NLP model_anno_obj sent_bert_use_cmlm_en_large\n",
            "nlu.load('en.embed_sentence.biobert.clinical_base_cased') returns Spark NLP model_anno_obj sent_biobert_clinical_base_cased\n",
            "nlu.load('en.embed_sentence.biobert.discharge_base_cased') returns Spark NLP model_anno_obj sent_biobert_discharge_base_cased\n",
            "nlu.load('en.embed_sentence.biobert.pmc_base_cased') returns Spark NLP model_anno_obj sent_biobert_pmc_base_cased\n",
            "nlu.load('en.embed_sentence.biobert.pubmed_base_cased') returns Spark NLP model_anno_obj sent_biobert_pubmed_base_cased\n",
            "nlu.load('en.embed_sentence.biobert.pubmed_large_cased') returns Spark NLP model_anno_obj sent_biobert_pubmed_large_cased\n",
            "nlu.load('en.embed_sentence.biobert.pubmed_pmc_base_cased') returns Spark NLP model_anno_obj sent_biobert_pubmed_pmc_base_cased\n",
            "nlu.load('en.embed_sentence.covidbert.large_uncased') returns Spark NLP model_anno_obj sent_covidbert_large_uncased\n",
            "nlu.load('en.embed_sentence.distil_roberta.distilled_base') returns Spark NLP model_anno_obj sent_distilroberta_base\n",
            "nlu.load('en.embed_sentence.doc2vec') returns Spark NLP model_anno_obj doc2vec_gigaword_300\n",
            "nlu.load('en.embed_sentence.doc2vec.gigaword_300') returns Spark NLP model_anno_obj doc2vec_gigaword_300\n",
            "nlu.load('en.embed_sentence.doc2vec.gigaword_wiki_300') returns Spark NLP model_anno_obj doc2vec_gigaword_wiki_300\n",
            "nlu.load('en.embed_sentence.electra') returns Spark NLP model_anno_obj sent_electra_small_uncased\n",
            "nlu.load('en.embed_sentence.electra_base_uncased') returns Spark NLP model_anno_obj sent_electra_base_uncased\n",
            "nlu.load('en.embed_sentence.electra_large_uncased') returns Spark NLP model_anno_obj sent_electra_large_uncased\n",
            "nlu.load('en.embed_sentence.electra_small_uncased') returns Spark NLP model_anno_obj sent_electra_small_uncased\n",
            "nlu.load('en.embed_sentence.roberta.base') returns Spark NLP model_anno_obj sent_roberta_base\n",
            "nlu.load('en.embed_sentence.roberta.large') returns Spark NLP model_anno_obj sent_roberta_large\n",
            "nlu.load('en.embed_sentence.small_bert_L10_128') returns Spark NLP model_anno_obj sent_small_bert_L10_128\n",
            "nlu.load('en.embed_sentence.small_bert_L10_256') returns Spark NLP model_anno_obj sent_small_bert_L10_256\n",
            "nlu.load('en.embed_sentence.small_bert_L10_512') returns Spark NLP model_anno_obj sent_small_bert_L10_512\n",
            "nlu.load('en.embed_sentence.small_bert_L10_768') returns Spark NLP model_anno_obj sent_small_bert_L10_768\n",
            "nlu.load('en.embed_sentence.small_bert_L12_128') returns Spark NLP model_anno_obj sent_small_bert_L12_128\n",
            "nlu.load('en.embed_sentence.small_bert_L12_256') returns Spark NLP model_anno_obj sent_small_bert_L12_256\n",
            "nlu.load('en.embed_sentence.small_bert_L12_512') returns Spark NLP model_anno_obj sent_small_bert_L12_512\n",
            "nlu.load('en.embed_sentence.small_bert_L12_768') returns Spark NLP model_anno_obj sent_small_bert_L12_768\n",
            "nlu.load('en.embed_sentence.small_bert_L2_128') returns Spark NLP model_anno_obj sent_small_bert_L2_128\n",
            "nlu.load('en.embed_sentence.small_bert_L2_256') returns Spark NLP model_anno_obj sent_small_bert_L2_256\n",
            "nlu.load('en.embed_sentence.small_bert_L2_512') returns Spark NLP model_anno_obj sent_small_bert_L2_512\n",
            "nlu.load('en.embed_sentence.small_bert_L2_768') returns Spark NLP model_anno_obj sent_small_bert_L2_768\n",
            "nlu.load('en.embed_sentence.small_bert_L4_128') returns Spark NLP model_anno_obj sent_small_bert_L4_128\n",
            "nlu.load('en.embed_sentence.small_bert_L4_256') returns Spark NLP model_anno_obj sent_small_bert_L4_256\n",
            "nlu.load('en.embed_sentence.small_bert_L4_512') returns Spark NLP model_anno_obj sent_small_bert_L4_512\n",
            "nlu.load('en.embed_sentence.small_bert_L4_768') returns Spark NLP model_anno_obj sent_small_bert_L4_768\n",
            "nlu.load('en.embed_sentence.small_bert_L6_128') returns Spark NLP model_anno_obj sent_small_bert_L6_128\n",
            "nlu.load('en.embed_sentence.small_bert_L6_256') returns Spark NLP model_anno_obj sent_small_bert_L6_256\n",
            "nlu.load('en.embed_sentence.small_bert_L6_512') returns Spark NLP model_anno_obj sent_small_bert_L6_512\n",
            "nlu.load('en.embed_sentence.small_bert_L6_768') returns Spark NLP model_anno_obj sent_small_bert_L6_768\n",
            "nlu.load('en.embed_sentence.small_bert_L8_128') returns Spark NLP model_anno_obj sent_small_bert_L8_128\n",
            "nlu.load('en.embed_sentence.small_bert_L8_256') returns Spark NLP model_anno_obj sent_small_bert_L8_256\n",
            "nlu.load('en.embed_sentence.small_bert_L8_512') returns Spark NLP model_anno_obj sent_small_bert_L8_512\n",
            "nlu.load('en.embed_sentence.small_bert_L8_768') returns Spark NLP model_anno_obj sent_small_bert_L8_768\n",
            "nlu.load('en.embed_sentence.tfhub_use') returns Spark NLP model_anno_obj tfhub_use\n",
            "nlu.load('en.embed_sentence.tfhub_use.lg') returns Spark NLP model_anno_obj tfhub_use_lg\n",
            "nlu.load('en.embed_sentence.use') returns Spark NLP model_anno_obj tfhub_use\n",
            "nlu.load('en.embed_sentence.use.lg') returns Spark NLP model_anno_obj tfhub_use_lg\n",
            "For language <es> NLU provides the following Models : \n",
            "nlu.load('es.embed_sentence.bert.base_cased') returns Spark NLP model_anno_obj sent_bert_base_cased\n",
            "nlu.load('es.embed_sentence.bert.base_uncased') returns Spark NLP model_anno_obj sent_bert_base_uncased\n",
            "For language <fi> NLU provides the following Models : \n",
            "nlu.load('fi.embed_sentence.bert') returns Spark NLP model_anno_obj bert_base_finnish_uncased\n",
            "nlu.load('fi.embed_sentence.bert.cased') returns Spark NLP model_anno_obj bert_base_finnish_cased\n",
            "nlu.load('fi.embed_sentence.bert.uncased') returns Spark NLP model_anno_obj bert_base_finnish_uncased\n",
            "For language <ha> NLU provides the following Models : \n",
            "nlu.load('ha.embed_sentence.xlm_roberta') returns Spark NLP model_anno_obj sent_xlm_roberta_base_finetuned_hausa\n",
            "For language <ig> NLU provides the following Models : \n",
            "nlu.load('ig.embed_sentence.xlm_roberta') returns Spark NLP model_anno_obj sent_xlm_roberta_base_finetuned_igbo\n",
            "For language <lg> NLU provides the following Models : \n",
            "nlu.load('lg.embed_sentence.xlm_roberta') returns Spark NLP model_anno_obj sent_xlm_roberta_base_finetuned_luganda\n",
            "For language <nl> NLU provides the following Models : \n",
            "nlu.load('nl.embed_sentence.bert.base_cased') returns Spark NLP model_anno_obj sent_bert_base_cased\n",
            "For language <pcm> NLU provides the following Models : \n",
            "nlu.load('pcm.embed_sentence.xlm_roberta') returns Spark NLP model_anno_obj sent_xlm_roberta_base_finetuned_naija\n",
            "For language <pt> NLU provides the following Models : \n",
            "nlu.load('pt.embed_sentence.bert.base_legal') returns Spark NLP model_anno_obj sbert_legal_bertimbau_base_tsdae_sts\n",
            "nlu.load('pt.embed_sentence.bert.cased_large_legal') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_sts_v0.1\n",
            "nlu.load('pt.embed_sentence.bert.large_legal') returns Spark NLP model_anno_obj sbert_legal_bertimbau_large_gpl_sts\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_sts_v0.10.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_sts_v0.10\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_sts_v0.2.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_sts_v0.2\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_sts_v0.3.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_sts_v0.3\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_sts_v0.4.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_sts_v0.4\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_sts_v0.5.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_sts_v0.5\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_sts_v0.7.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_sts_v0.7\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_sts_v0.8.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_sts_v0.8\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_sts_v0.9.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_sts_v0.9\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_sts_v1.0.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_sts_v1.0\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_v0.11_gpl_nli_sts_v0.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_v0.11_gpl_nli_sts_v0\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_v0.11_gpl_nli_sts_v1.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_v0.11_gpl_nli_sts_v1\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_v0.11_nli_sts_v0.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_v0.11_nli_sts_v0\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_v0.11_nli_sts_v1.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_v0.11_nli_sts_v1\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_v0.11_sts_v0.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_v0.11_sts_v0\n",
            "nlu.load('pt.embed_sentence.bert.legal.cased_large_mlm_v0.11_sts_v1.by_stjiris') returns Spark NLP model_anno_obj sbert_bert_large_portuguese_cased_legal_mlm_v0.11_sts_v1\n",
            "nlu.load('pt.embed_sentence.bert.v2_base_legal') returns Spark NLP model_anno_obj sbert_legal_bertimbau_sts_base_ma_v2\n",
            "nlu.load('pt.embed_sentence.bert.v2_large_legal') returns Spark NLP model_anno_obj sbert_legal_bertimbau_large_tsdae_sts_v2\n",
            "nlu.load('pt.embed_sentence.bertimbau.legal.assin.base.by_rufimelo') returns Spark NLP model_anno_obj sbert_legal_bertimbau_sts_base_ma\n",
            "nlu.load('pt.embed_sentence.bertimbau.legal.assin2.base.by_rufimelo') returns Spark NLP model_anno_obj sbert_legal_bertimbau_sts_base\n",
            "nlu.load('pt.embed_sentence.bertimbau.legal.large_sts_by_rufimelo') returns Spark NLP model_anno_obj sbert_legal_bertimbau_sts_large\n",
            "nlu.load('pt.embed_sentence.bertimbau.legal.large_sts_ma.by_rufimelo') returns Spark NLP model_anno_obj sbert_legal_bertimbau_sts_large_ma\n",
            "nlu.load('pt.embed_sentence.bertimbau.legal.large_sts_ma_v3.by_rufimelo') returns Spark NLP model_anno_obj sbert_legal_bertimbau_sts_large_ma_v3\n",
            "nlu.load('pt.embed_sentence.bertimbau.legal.large_tsdae_sts.by_rufimelo') returns Spark NLP model_anno_obj sbert_legal_bertimbau_large_tsdae_sts\n",
            "nlu.load('pt.embed_sentence.bertimbau.legal.large_tsdae_sts_v4.by_rufimelo') returns Spark NLP model_anno_obj sbert_legal_bertimbau_large_tsdae_sts_v4\n",
            "nlu.load('pt.embed_sentence.bertimbau.legal.large_tsdae_v4_gpl_sts.by_rufimelo') returns Spark NLP model_anno_obj sbert_legal_bertimbau_large_tsdae_v4_gpl_sts\n",
            "nlu.load('pt.embed_sentence.bertimbau.legal.v2_large_sts_v2.by_rufimelo') returns Spark NLP model_anno_obj sbert_legal_bertimbau_sts_large_v2\n",
            "nlu.load('pt.embed_sentence.bertimbau.legal.v2_large_v2_sts.by_rufimelo') returns Spark NLP model_anno_obj sbert_legal_bertimbau_large_v2_sts\n",
            "For language <rw> NLU provides the following Models : \n",
            "nlu.load('rw.embed_sentence.xlm_roberta') returns Spark NLP model_anno_obj sent_xlm_roberta_base_finetuned_kinyarwanda\n",
            "For language <sv> NLU provides the following Models : \n",
            "nlu.load('sv.embed_sentence.bert.base_cased') returns Spark NLP model_anno_obj sent_bert_base_cased\n",
            "For language <sw> NLU provides the following Models : \n",
            "nlu.load('sw.embed_sentence.xlm_roberta') returns Spark NLP model_anno_obj sent_xlm_roberta_base_finetuned_swahili\n",
            "For language <wo> NLU provides the following Models : \n",
            "nlu.load('wo.embed_sentence.xlm_roberta') returns Spark NLP model_anno_obj sent_xlm_roberta_base_finetuned_wolof\n",
            "For language <xx> NLU provides the following Models : \n",
            "nlu.load('xx.embed_sentence') returns Spark NLP model_anno_obj sent_bert_multi_cased\n",
            "nlu.load('xx.embed_sentence.bert') returns Spark NLP model_anno_obj sent_bert_multi_cased\n",
            "nlu.load('xx.embed_sentence.bert.cased') returns Spark NLP model_anno_obj sent_bert_multi_cased\n",
            "nlu.load('xx.embed_sentence.bert.muril') returns Spark NLP model_anno_obj sent_bert_muril\n",
            "nlu.load('xx.embed_sentence.bert_use_cmlm_multi_base') returns Spark NLP model_anno_obj sent_bert_use_cmlm_multi_base\n",
            "nlu.load('xx.embed_sentence.bert_use_cmlm_multi_base_br') returns Spark NLP model_anno_obj sent_bert_use_cmlm_multi_base_br\n",
            "nlu.load('xx.embed_sentence.labse') returns Spark NLP model_anno_obj labse\n",
            "nlu.load('xx.embed_sentence.xlm_roberta.base') returns Spark NLP model_anno_obj sent_xlm_roberta_base\n",
            "For language <yo> NLU provides the following Models : \n",
            "nlu.load('yo.embed_sentence.xlm_roberta') returns Spark NLP model_anno_obj sent_xlm_roberta_base_finetuned_yoruba\n",
            "For language <zh> NLU provides the following Models : \n",
            "nlu.load('zh.embed_sentence.bert') returns Spark NLP model_anno_obj sbert_chinese_qmc_finance_v1\n",
            "nlu.load('zh.embed_sentence.bert.distilled') returns Spark NLP model_anno_obj sbert_chinese_qmc_finance_v1_distill\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IKK_Ii_gjJfF",
        "outputId": "ae16a73d-eb90-4d1f-f1d0-9769c28ce54f"
      },
      "source": [
        "trainable_pipe = nlp.load('en.embed_sentence.small_bert_L12_768 train.sentiment')\n",
        "# We need to train longer and user smaller LR for NON-USE based sentence embeddings usually\n",
        "# We could tune the hyperparameters further with hyperparameter tuning methods like gridsearch\n",
        "# Also longer training gives more accuracy\n",
        "trainable_pipe['trainable_sentiment_dl'].setMaxEpochs(100)\n",
        "trainable_pipe['trainable_sentiment_dl'].setLr(0.0005)\n",
        "fitted_pipe = trainable_pipe.fit(train_df)\n",
        "# predict with the trainable pipeline on dataset and get predictions\n",
        "preds = fitted_pipe.predict(train_df,output_level='document')\n",
        "\n",
        "#sentence detector that is part of the pipe generates sone NaNs. lets drop them first\n",
        "preds.dropna(inplace=True)\n",
        "print(classification_report(preds['y'], preds['sentiment']))\n",
        "\n",
        "#preds"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Warning::Spark Session already created, some configs may not take.\n",
            "Warning::Spark Session already created, some configs may not take.\n",
            "sent_small_bert_L12_768 download started this may take some time.\n",
            "Approximate size to download 392.9 MB\n",
            "[OK!]\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "    negative       0.78      0.65      0.71       300\n",
            "     neutral       0.00      0.00      0.00         0\n",
            "    positive       0.89      0.52      0.65       300\n",
            "\n",
            "    accuracy                           0.58       600\n",
            "   macro avg       0.55      0.39      0.45       600\n",
            "weighted avg       0.83      0.58      0.68       600\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/nlu/pipe/extractors/extractor_methods/base_extractor_methods.py:356: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  df[cols_to_explode] = df[cols_to_explode].apply(pad_same_level_cols, axis=1)\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2BB-NwZUoHSe"
      },
      "source": [
        "# 5. Lets save the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eLex095goHwm"
      },
      "source": [
        "stored_model_path = './models/classifier_dl_trained'\n",
        "fitted_pipe.save(stored_model_path)"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e_b2DPd4rCiU"
      },
      "source": [
        "# 6. Lets load the model from HDD.\n",
        "This makes Offlien NLU usage possible!   \n",
        "You need to call nlu.load(path=path_to_the_pipe) to load a model/pipeline from disk."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SO4uz45MoRgp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 133
        },
        "outputId": "5634381d-e25b-49dd-d720-860be456d9cd"
      },
      "source": [
        "hdd_pipe = nlp.load(path=stored_model_path)\n",
        "\n",
        "preds = hdd_pipe.predict('the president of india just died')\n",
        "preds"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Warning::Spark Session already created, some configs may not take.\n",
            "Warning::Spark Session already created, some configs may not take.\n",
            "Warning::Spark Session already created, some configs may not take.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                           document  \\\n",
              "0  the president of india just died   \n",
              "\n",
              "                        sentence_embedding_from_disk sentiment  \\\n",
              "0  [0.009459968656301498, -0.07943318039178848, 0...  positive   \n",
              "\n",
              "  sentiment_confidence  \n",
              "0                  0.0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8f136e86-5df7-4581-9143-cf2be51c2c2f\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>document</th>\n",
              "      <th>sentence_embedding_from_disk</th>\n",
              "      <th>sentiment</th>\n",
              "      <th>sentiment_confidence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>the president of india just died</td>\n",
              "      <td>[0.009459968656301498, -0.07943318039178848, 0...</td>\n",
              "      <td>positive</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8f136e86-5df7-4581-9143-cf2be51c2c2f')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-8f136e86-5df7-4581-9143-cf2be51c2c2f button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-8f136e86-5df7-4581-9143-cf2be51c2c2f');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e0CVlkk9v6Qi",
        "outputId": "54c80119-8661-4b08-f9c3-952193759535"
      },
      "source": [
        "hdd_pipe.print_info()"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The following parameters are configurable for this NLU pipeline (You can copy paste the examples) :\n",
            ">>> component_list['document_assembler'] has settable params:\n",
            "component_list['document_assembler'].setCleanupMode('shrink')                                    | Info: possible values: disabled, inplace, inplace_full, shrink, shrink_full, each, each_full, delete_full | Currently set to : shrink\n",
            ">>> component_list['bert_sentence_embeddings@sent_small_bert_L12_768'] has settable params:\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L12_768'].setBatchSize(8)               | Info: Size of every batch | Currently set to : 8\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L12_768'].setCaseSensitive(False)       | Info: whether to ignore case in tokens for embeddings matching | Currently set to : False\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L12_768'].setDimension(768)             | Info: Number of embedding dimensions | Currently set to : 768\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L12_768'].setMaxSentenceLength(128)     | Info: Max sentence length to process | Currently set to : 128\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L12_768'].setEngine('tensorflow')       | Info: Deep Learning engine used for this model | Currently set to : tensorflow\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L12_768'].setIsLong(False)              | Info: Use Long type instead of Int type for inputs buffer - Some Bert models require Long instead of Int. | Currently set to : False\n",
            "component_list['bert_sentence_embeddings@sent_small_bert_L12_768'].setStorageRef('sent_small_bert_L12_768')  | Info: unique reference name for identification | Currently set to : sent_small_bert_L12_768\n",
            ">>> component_list['sentiment_dl@sent_small_bert_L12_768'] has settable params:\n",
            "component_list['sentiment_dl@sent_small_bert_L12_768'].setThreshold(0.6)                         | Info: The minimum threshold for the final result otheriwse it will be neutral | Currently set to : 0.6\n",
            "component_list['sentiment_dl@sent_small_bert_L12_768'].setThresholdLabel('neutral')              | Info: In case the score is less than threshold, what should be the label. Default is neutral. | Currently set to : neutral\n",
            "component_list['sentiment_dl@sent_small_bert_L12_768'].setEngine('tensorflow')                   | Info: Deep Learning engine used for this model | Currently set to : tensorflow\n",
            "component_list['sentiment_dl@sent_small_bert_L12_768'].setClasses(['positive', 'negative'])      | Info: get the tags used to trained this SentimentDLModel | Currently set to : ['positive', 'negative']\n",
            "component_list['sentiment_dl@sent_small_bert_L12_768'].setStorageRef('sent_small_bert_L12_768')  | Info: unique reference name for identification | Currently set to : sent_small_bert_L12_768\n"
          ]
        }
      ]
    }
  ]
}